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(54) IMAGE RETRIEVING METHOD 

(57)Abstract: 

PROBLEM TO BE SOLVED: To make easily retrievable 
an image by specifying, as reference data, the 
photographing place, the position or name of a subject 
and finding to what extent the data match the 
information attached to the data of a stored image. 
SOLUTION: The direction and photographic power of a 
camera at the time of photographing are superposed on 
map information and a photographed subject is specified 
and stored in a database in sequence. A specification 
pattern for retrieval is read in and superposed on the 
map information to perform conversion to retrieval 
specification information of a user, i.e., positional 
information on the subject. Then, an original image 
database is read in and the read information is collected 
with tabled specific patterns for retrieval. Here, the 
columns and rows of the table are collated sequentially 
and matching columns or rows, when found, are taken 
out as a result of retrieval. Then the degree of matching 
(similarity) with a plurality of candidate subjects in the 
original image database is calculated and retrieved images are outputted in the decreasing order 
of the points of the degree of matching. 
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* NOTICES * 

JPO and NCI PI are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] The location on the name of a photography location or a photographic subject and a 
map is extracted from the photography information and map information at the time of 
photography. It is made attached to image data by making an extract result into attached 
information, and memorizes. The location or name of a photography location, Or the image 
search method characterized by reading the image which specifies the location or name of a 
photographic subject as reference data, asks for whenever [ with the attached information 
attached to the data of said image made to memorize / coincidence ], and has whenever 
[ beyond a predetermined value / coincidence ]. 

[Claim 2] The image search method according to claim 1 with which said photography 
information consists of the 1st data configuration which it becomes from a camera station, the 
2nd data configuration which consists of a camera station and photography bearing, the 3rd data 
configuration which consists of a camera station and a photography scale factor, the camera 
station and photography bearing which are acquired by relating with a photography image, and a 
photography scale factor and which is either of the 4th data configuration. 
[Claim 3] An image search method given in either of claims 1 or 2 which specify the outline of 
depth by arranging the photographic subject patternized to virtual map space as reference data 
about the location and photographic subject location of said photography. 

[Claim 4] An image search method given in any 1 term of claims 1-3 which add to said extract 
information, extract the location and size in a photography location or the screen of a 
photographic subject, and add this to said attached information. 

[Claim 5] An image search method given in any 1 term of claims 1~3 which add to said extract 
information and apply the location and size in a photography location or the screen of a 
photographic subject to assignment information. 

[Claim 6] The image search method which draws the distance information on a photography 
screen from the positional information, the camera information, and the map information at the 
time of photography, and carries out [ reading the image which attaches to image data, 
memorizes, specifies the distance information on a photography screen as reference data, asks 
for whenever / with the attached information attached to the data of said image made to 
memorize / coincidence /, and has whenever / beyond a predetermined value / coincidence /, 
and ] as the description by making a derivation result into attached information. 
[Claim 7] The image search method according to claim 6 which roughly specifies the depth 
according to field in a photography screen as reference data about the distance information on 
said photography screen. 

[Claim 8] The image search method according to claim 6 which specifies the outline of depth by 
arranging the photographic subject patternized to virtual map space as reference data about the 
distance information on said photography screen. 

[Claim 9] An image search method given in any 1 term of claims 1~8 which remember difference 
with an actual photography image to be the simulation image data of the photography image 
which created from the positional information, the camera information, and the map information 
at the time of photography, and are characterized by to output the image which added said 



http://www4.ipdl.ncipi.go jp/cgi-bin/tran_web_cgLeiie?u=http%3A%2F%2Fwww4.ipdl.... 2006/06/08 



« JP.20Q1 -034632.A [CLAIMS] 



2/2 ^— is 



difference made to memorize to the simulation image data of the photography image created 
from the positional information, the camera information, and the map information at the time of 
said photography at the time of the image output of a retrieval result. 



[Translation done.] 
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1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] About an image search method, a photographic subject extracts 
something from the positional information, camera information, and map information at the time 
of photography, and, more specifically, this invention relates to the image search method which 
used this extract image for retrieval. 
[0002] 

[Description of the Prior Art] Kamiichi of the camera which whether "it is where when" 
photography of the photograph was carried out can record automatically in order to present 
photography of a record photograph in recent years is carried out. This camera is the synthetic 
goods with which the location metering device GPS (Global Positioninng System) using the 
satellite which car navigation (car-navigation system) is sufficient as, and is used and the bearing 
sensor, and the camera were put together (photograph industry, 1996-7, pp.1 2-1 5, and 84~87). 
[0003] In the role of photography, the importance of the recordability of a photograph and 
testimony nature is one of the essential value which a photograph has. Therefore, in a record 
photograph, the incidental information which shows whether "it is where when" photography was 
carried out plays an important role about a photographic subject. Although the conventional 
camera had offered the information of "when" to some extent, the information of "where" was 
not offered. 

[0004] The camera (henceforth a camera with built-in GPS) currently introduced to the above- 
mentioned photograph industrial magazine can be called what achieves the function of record 
photograph original in this semantics. That is, first, to the difference between the so-called date 
function of the conventional camera with a date function (former) and the time stamp of a 
camera with built-in GPS being the time of day which the former can determine privately so to 
speak, the latter is the point which is transmitted from a satellite and which is "time of day of a 
formula" so to speak, and differ deterministically in respect of calling it testimony nature. 
[0005] On the other hand, the positional information which a camera with built-in GPS shows is 
not the location of a photographic subject but the location of a camera, and the direction of 
photography correctly. For this reason, when a photograph is a distant view, the location of a 
photographic subject is not necessarily clear. Also in this case, it is possible to compute the 
location of a photographic subject correctly from the distance between the photographic 
subjects of the plurality in an image etc. as occasion demands. 
[0006] 

[Problem(s) to be Solved by the Invention] By the way, this invention person related with the 
photoed image previously, acquired at least one photography information on a camera station, 
photography bearing, and a photography scale factor, and has proposed the photoprint equipment 
which materialized the photoprint approach which acquires the additional information relevant to 
a part of this acquired photography information [ at least ], and added the acquired additional 
information to the photoprint, and this approach by Japanese Patent Application No. No. 162130 
[ ten to ]. 

[0007] Since according to the above-mentioned photoprint approach and equipment a customer 
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can see an image and can recollect the situation at the time of photography etc. easily based on 
the photography information added to the photoed image by relating, it is effective also at the 
point it not only raises the memorandum-function of a photoprint, but that it can give 
enjoyableness (you may call it subject offer nature) to a photoprint. 
[0008] In case a customer searches a desired image out of the image of a large number 
accumulated, the place which this invention was made in order to make the above-mentioned 
technique further easy to use, and makes into the purpose is using the positional information, the 
camera information, and the map information at the time of the photography added to the 
photoed above-mentioned image by relating, and is to offer the image search method which 
enables retrieval of an image easily. 
[0009] 

[Means for Solving the Problem] In order to attain the above-mentioned purpose, the image 
search method concerning this invention The location on the name of a photography location or 
a photographic subject and a map is extracted from the photography information and map 
information at the time of photography. It is made attached to image data by making an extract 
result into attached information, and memorizes. The location or name of a photography location, 
Or the location or name of a photographic subject is specified as reference data, and it asks for 
whenever [ with the attached information attached to the data of said image made to 
memorize / coincidence ], and is characterized by reading the image which has whenever 
[ beyond a predetermined value / coincidence ]. 

[0010] In the image search method concerning this invention, it is characterized by for said 
photography information to consist of the 1st data configuration which it becomes from a 
camera station, the 2nd data configuration which consists of a camera station and photography 
bearing, the 3rd data configuration which consists of a camera station and a photography scale 
factor, the camera station and photography bearing which are acquired by relating with a 
photography image, and a photography scale factor in it being either of the 4th data 
configuration. 

[001 1] In the image search method concerning this invention, it is characterized by specifying 
the outline of depth by arranging the photographic subject patternized to virtual map space as 
reference data about the location and photographic subject location of said photography. 
Moreover, it adds to said extract information, the location and size in a photography location or 
the screen of a photographic subject are extracted, and it is characterized by adding this to said 
attached information. 

[0012] Moreover, in the image search method concerning this invention, it is also desirable to 
add to said extract information and to apply the location and size in a photography location or 
the screen of a photographic subject to assignment information. 

[0013] Moreover, the image search method concerning this invention draws the distance 
information on a photography screen from the positional information, camera information, and 
map information at the time of photography. By making a derivation result into attached 
information, it is made attached to image data, and memorizes, the distance information on a 
photography screen is specified as reference data, and it asks for whenever [ with the attached 
information attached to the data of said image made to memorize / coincidence ], and is 
characterized by reading the image which has whenever [ beyond a predetermined value / 
coincidence ]. 

[0014] In the image search method concerning this invention, it is characterized by roughly 
specifying the depth according to field in a photography screen as reference data about the 
distance information on said photography screen. Moreover, it is characterized by specifying the 
outline of depth by arranging the photographic subject patternized to virtual map space as 
reference data about the distance information on said photography screen. 
[0015] Moreover, it sets to the image search method concerning this invention. Difference with 
an actual photography image is remembered to be the simulation image data of the photography 
image created from the positional information, camera information, and map information at the 
time of photography. At the time of the image output of a retrieval result It is desirable to reduce 
stored data capacity by outputting the image which added said difference made to memorize to 



http://www4.ipdl.ncipi.go jp/cgi-bin/tran_web_cgi_ejje 



2006/06/08 



JP,20Q1-034632,A [DETAILED DESCRIPTION] 



3/6 ^— v> 



the simulation image data of the photography image created from the positional information, 

camera information, and map information at the time of said photography. 

[0016] 

[Embodiment of the Invention] Hereafter, the gestalt of operation of this invention is explained to 
a detail based on the suitable example shown in an attached drawing. 

[0017] Drawing 1 and drawing 2 are drawings for explaining the image search method concerning 
one example of this invention, and in case drawing 2 searches this subject-copy image 10 for the 
subject-copy image 10 on the film by which drawing 1 was photoed with the camera with a GPS 
function, the user shows the assignment pattern 20 for retrieval inputted using the mouse etc. 
on the screen of a terminal unit. 

[0018] The prominent building (it is called ** and x building) whose 13 is a photographic subject 
also with this effective in pinpointing of a camera station about the crest (it is hereafter called O 
crest for convenience) which is a photographic subject effective in pinpointing of the camera 
station where 12 mentions later the person whose 1 1 is a main photographic subject in the 
subject-copy image 10 shown in drawing 1 is shown. 

[0019] moreover, the assignment pattern 20 for retrieval shown in drawing 2 — a user — for 
example, — " — the T 1 area neighborhood — somehow — ** — the crest say — it is — the T 
2 area neighborhood — somehow — ** — I wants to search the scene which has the building 
say — " — ** — it is the example which expressed and inputted the hope which says as an 
assignment pattern for retrieval, and they are the display whose 22 shows a crest's existence 
location, and the display 23 indicates a building s existence location to be 

[0020] It explains according to the flow Fig. of operation which shows the outline which operates 
the location of the photographic subject which the location in the subject-copy image 10 
hereafter shown in drawing 1 as housekeeping operation in the image search method concerning 
this example first can specify, and specification of a name to drawing 3 . 
[0021] As shown in drawing 3 , when an image is read, the GPS information at the above- 
mentioned photography time added to this, and the direction and photography scale factor of a 
camera at the time of photography are also read (step 101). A photography point is first 
pinpointed to the map information currently prepared beforehand by piling up the above- 
mentioned GPS information (step 102). 

[0022] Next, the photographic subject currently photoed is specified as map information by piling 
up the direction and photography scale factor of a camera at the time of the above-mentioned 
photography (step 103). This processing creates a simulation image based on the direction and 
photography scale factor of a camera in the applicable location on a map, and you may make it 
specify a photographic subject with pattern matching with a photography image. Each specified 
photographic subject is memorized by the database one by one (step 104). 
[0023] The example of the attached information on the image data of the subject-copy image 
created at the above-mentioned subject-copy image based on the camera station information 
added as additional information is shown in Table 1. As shown in Table 1, the aforementioned O 
crest, x building, etc. which the location understands beforehand are recorded [ here ] and 
selected with the location on a screen, the location on a configuration and a map, and the name 
by the direction of the GPS information at the photography time, and the camera at the time of 
photography etc. In addition, it registers with a database as attached information on a 
photography image also about photography time, camera station information (the LAT, LONG, 
name of a place), and other camera information. 
[0024] 
[Table 1] 
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[0025] In addition, in this example, since the above-mentioned data **** step is performed 
automatically, in case image data is registered into a database, the effectiveness that the time 
and effort which inputs a photographic subject name as data for retrieval can be saved is 
acquired. 

[0026] The user who is going to search an image inputs positional information (LAT, LONG), the 
name of a place, or a name of a photographic subject in a screen etc. as retrieval information. 
This input may be performed using a mouse etc. on a screen of a terminal unit like the 
assignment pattern 20 for retrieval shown in drawing 2 . In this case, a location will be indirectly 
specified by specifying the identifier of a crest or a building. 

[0027] Next, the flow of the actuation in this example which searches an image based on the 
data memorized as mentioned above is explained based on drawing 4 . 

[0028] First, the assignment pattern for retrieval as shown in drawing 4 is read (step 201), and it 
changes into a users assignment information for retrieval, i.e., the positional information of a 
photographic subject, by piling this up with map information (step 202). In addition, these steps 
may be flown when a user inputs positional information. 

[0029] Next, a subject-copy image database is read (step 203), and it collates with the table- 
ized assignment pattern for retrieval (step 204). If this collating performs each train of a table, 
and each line one by one and there is a match, it will take out as a retrieval result (steps 205- 
209). 

[0030] In addition, since it is rare to be completely in agreement in fact, whenever 
[ coincidence ] is defined and it is good [ it explained "it will take out if there is a match" here 
when collating in order to simplify explanation, but ] that it should be in agreement in that to 
which whenever [ coincidence / which computed ] exceeded the predetermined threshold so 
that it may mention later. 

[0031] As an above-mentioned count method of whenever [ coincidence ], with the method 
using the information of the main coordinate in the attached information shown in Table 1, size, 
and the direction of top and bottom, whenever [ coincidence ] (or similarity) is calculated about 
two or more candidate photographic subjects in a subject-copy image database, and the 
searched image is outputted sequentially from the large thing of the point of whenever [ of 
these / coincidence ], for example. 

[0032] According to the image search method shown in this example, it enables a user to search 
the image which suits directions based on the assignment pattern 20 for retrieval inputted using 
the mouse etc. on the screen of a terminal unit etc., positional information (LAT, LONG), the 
name of a place or a name of a photographic subject in the screen inputted with the keyboard as 
retrieval information, etc. 

[0033] Next, other examples of this invention are explained. Although the above-mentioned 
example showed the example using a subject-copy image and the database created from the 
photography information, this example explains the case where distinction or collating of a scene 
is made to be performed, using the depth information in an image. In addition, this example is the 
case where neither camera station information nor map information is used. Here, the 
information on the above-mentioned depth is good to use what was drawn from the dynamic- 
focusing image / stereo image which is a well-known camera technique. 
[0034] An example of the retrieval assignment pattern 30 which includes the above depth 
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information in drawing 5 is shown. In the example shown in drawing 5 , the round mark of 31 is a 
person, it is made an upper distance, a distant place is equivalent to a crest (oblong ellipse form 
32), and four right-hand side square shapes 33 are equivalent to a building. 

[0035] In the example shown in drawing 5 , it sees from a camera, a person 31 is in the distance 
which is about 5-1 0m, a building 33 exists in the back about 100m of right-hand side, and, 
specifically, the sight in which a crest 32 exists about 1000m behind a person is expressed 
further. In addition, it is also possible to use combining assignment of the location of the 
photographic subject in the screen shown in drawing 2 and arrangement by the configuration of 
the photographic subject shown in drawing 5 . 

[0036] With a plot plan as shown in drawing 5 , if a user specifies the image which should express 
the depth for every rough area of a scene, and should be searched, he will calculate whenever 
[ coincidence ] about these contents and two or more candidate photographic subjects in said 
subject-copy image database, and he outputs the searched image sequentially from the large 
thing of the point of whenever [ of these / coincidence ]. 

[0037] According to the image search method shown in this example, when the user added the 
arrangement situation of a superficial photographic subject to the assignment pattern 20 for 
retrieval to input, it is effective in coming to be able to perform more easily searching the image 
which suits directions. In addition, the image search method concerning this example can be 
applied also to an animation. In this case, the motion information between the coma of a dynamic 
image can also be used. 

[0038] Next, the example of further others of this invention is explained. In this example, the 
case where there is a plot plan showing the arrangement situation between photographic 
subjects is explained. Here, as the above-mentioned plot plan, as shown in drawing 5 , the 
superficial physical relationship of the crest and building to the person who is easy to be a main 
photographic subject should just be known, for example. 

[0039] Here, as positional information, the location judging (in this case, the precision of a 
location judging is very inferior to GPS in the present condition) by above-mentioned GPS 
information or PHS (Personal Handyphone System) or rougher tab-control-specification 
information (as [ call / for example, / it / eye xx city OO town **** ** ]) can be used. 
[0040] Then, whenever [ coincidence ] is computed for every candidate by performing matching 
processing by collating with the plot plan used for the arrangement and assignment on map data. 
In addition, when there is no bearing information on a camera, simulation is performed by rotating 
camera bearing for every predetermined include angle (0-360 degrees). Moreover, when the 
precision of a location is low, simulation is performed by carrying out a parallel displacement by 
predetermined within the limits. 

[0041] According to this example, it becomes possible to search the image using the information 
on geographical feature etc. with easy directions, and the effectiveness of becoming possible to 
calculate the increase in efficiency of image retrieval is acquired. 

[0042] By the way, when the above images were searched conventionally, positional information 
and an eye direction were given in a form like drawing 6 (b) in many cases. That is, they are a 
camera location (GPS information) and bearing of the exposure axis (camera information). On the 
other hand, if the information on a form like drawing 6 (a), i.e., a photographic subject location 
and bearing of the exposure axis, is acquired, the following will be possible based on this. 
[0043] That is, since I hear that it is turned out what is photoed, the image of the photographic 
subject currently photoed is taken out out of the image database currently prepared beforehand, 
and a computer graphic image is created based on this, and by the comparison with an actual 
photography image, after performing alignment, size doubling, etc., both difference is taken, and it 
compresses suitably if needed, and considers as stored data. 

[0044] What combined the image considered to be the real image photoed before and to 
approximate most using positional information and camera photography information as an image 
for simulation besides an above-mentioned computer graphic image, or this and an above- 
mentioned computer graphic image is available. Also when using which image, it is possible to 
reduce the amount of stored data. 

[0045] In addition, each of each above-mentioned examples shows an example of this invention, 
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and it cannot be overemphasized that this invention is not what should be limited to these. 
[0046] 

[Effect of the Invention] As mentioned above, as explained to the detail, in case a customer 
searches a desired image out of the image of a large number accumulated according to this 
invention, the effectiveness that the image search method whose retrieval of an image is enabled 
easily is realizable by using the positional information, camera information, and map information 
at the time of the photography added to the photoed above-mentioned image by relating is done 
so. 

[0047] That is, according to the image search method concerning this invention, it enables a user 
to search the image which suits directions based on the assignment pattern for retrieval inputted 
using the mouse etc. on the screen of a terminal unit etc., positional information (LAT, LONG), 
the name of a place or a name of a photographic subject in the screen inputted with the 
keyboard as retrieval information, etc. 

[0048] Moreover, when a user adds the arrangement situation of a superficial photographic 
subject to the assignment pattern for retrieval to input, it is effective in becoming possible more 
easily to search the image which suits directions. In addition, the image search method 
concerning this invention can be fundamentally applied also to an animation. 



[Translation done.] 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is drawing showing the subject-copy image 10 on the film photoed with the camera 
with a GPS function for explaining the image search method concerning one example of this 
invention. 

[Drawing 2] In case the subject-copy image 10 shown in drawing 1 is searched, it is drawing 
showing an example of the assignment pattern 20 for retrieval which the user inputted. 
[Drawing 3] It is the flow Fig. of operation showing the outline of operation of specifying the 
location and name of a photographic subject which the location in the subject-copy image 10 as 
housekeeping operation in the image search method concerning one example can specify. 
[Drawing 4] It is the flow Fig. [ of operation ] of operation in the image search method 
concerning one example with which an image is searched based on the memorized data and in 
which flowing and showing **. 

[Drawing 5] It is drawing showing an example of a retrieval assignment pattern including depth 
information concerning other examples. 

[Drawing 6] It is drawing showing the relation of positional information and an eye direction, and 
when it understands (a) what a photographic subject is, the case where, as for what it is, a 
photographic subject is not known only by turning out where (b) was photoed is shown. 
[Description of Notations] 

10 Subject-Copy Image 

1 1 Person 

1 2 Crest (O Crest) 

13 Building (X Building) 

20 Assignment Pattern for Retrieval 

22 Display Which Shows Crest's Existence Location 

23 Display Which Shows Building's Existence Location 

30 Retrieval Assignment Pattern 

31 Person (Corresponding Round Mark) 

32 Crest — Oblong Ellipse Form 

33 Building (Four Corresponding Square Shapes) 



[Translation done.] 
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* NOTICES * 

JPO and NCI PI are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DRAWINGS 



[Drawing 11 




[Drawing 2] 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 
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[Procedure revision] 

[Filing Date] September 9, Heisei 16 (2004. 9.9) 

[Procedure amendment 1] 

[Document to be Amended] Specification 

[Item(s) to be Amended] 0018 

[Method of Amendment] Modification 

[The contents of amendment] 

[0018] 

The prominent building (it is called ** and x building) whose 13 is a photographic subject also 
with this effective in pinpointing of a photography location or a camera station about the crest (it 
is hereafter called O crest for convenience) which is a photographic subject effective in 
pinpointing of a photography location or a camera station whose 12 mentions later the person 
whose 11 is a main photographic subject in the subject-copy image 10 shown in drawing 1 is 
shown. 

[Procedure amendment 2] 

[Document to be Amended] Specification 

[Item(s) to be Amended] 0020 

[Method of Amendment] Modification 

[The contents of amendment] 

[0020] 

It explains according to the flow Fig. of operation which shows the outline of operation of 
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specifying the location and name of a photographic subject which the location in the subject- 
copy image 10 hereafter shown in drawing 1 as housekeeping operation in the image search 
method concerning this example first can specify to drawing 3. 
[Procedure amendment 3] 
[Document to be Amended] Specification 
[Item(s) to be Amended] 0028 
[Method of Amendment] Modification 
[The contents of amendment] 
[0028] 

First the assignment pattern for retrieval as shown in drawing 2 is read (step 201), and it 

changes into a users assignment information for retrieval, i.e., the positional information of a 

photographic subject, by piling this up with map information (step 202). 

In addition, these steps may be flown when a user inputs positional information. 

[Procedure amendment 4] 

[Document to be Amended] Specification 

[Item(s) to be Amended] Explanation of a sign 

[Method of Amendment] Modification 

[The contents of amendment] 

[Description of Notations] 

10 Subject-Copy Image 

1 1 Person 

1 2 Crest (O Crest) 

13 Building (X Building) 

20 Assignment Pattern for Retrieval 

22 Display Which Shows Crest's Existence Location 

23 Display Which Shows Building's Existence Location 

30 Retrieval Assignment Pattern 

31 Person (Corresponding Round Mark) 

32 Crest (Oblong Corresponding Ellipse Form) 

33 Building (Four Corresponding Square Shapes) 



[Translation done.] 
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